Periodic Stepsize Adaptation

Previously, Bottou and LeCun [1] established that the second-order stochastic gradient descent (SGD) method can potentially achieve generalization performance as well as empirical optimum in a single pass through the training examples. However, second-order SGD requires computing the inverse of the Hessian matrix of the loss function, which is usually prohibitively expensive. Recently, we invented a new second-order SGD method, called Periodic Stepsize Adaptation (PSA). PSA explores a simple linear relation between the Hessian matrix and the Jacobian matrix of the mapping function. Instead of approximating Hessian, PSA approximates the Jacobian matrix which is proved to be simpler and more effective than approximating Hessian in an on-line setting. Experimental results for conditional random fields (CRF) and neural networks (NN) show that single-pass performance of PSA is very close to empirical optimum.